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PREFACE 


This  report  was  created  for  the  F-16  Aircrew  Training  De¬ 
velopment  Project  contract  no.  F02604-79-C8875  for  the  Tactical 
Air  Command  to  comply  with  the  requirements  of  CDRL  no.  B045. 

The  project  entailed  the  design  and  develofxnent  of  an  instruc¬ 
tional  system  for  the  P-16  RTU  and  instructor  pilots.  During  the 
course  of  the  project «  a  series  of  development  reports  was  issued 
describing  processes  and  products.  A  list  of  those  reports 
follows  this  page.  The  user  is  referred  to  Report  No.  34,  A 
Users  Guide  to  the  F-16  Training  Development  Reports,  for  an 
overview  and  explanation  of  the  series,  and  Report  No.  35,  F-16 
Final  Report,  for  an  overview  of  th^  Instructional  System 
Development  Project. 
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EXECUTIVE  SUMMARY 


The  purpose  of  this  report  is  to  list  and  discuss  the  con¬ 
straints  and  limits  of  the  various  instructional  media  employed 
in  F-16  training,  with  the  purpose  of  targeting  these  limita¬ 
tions  for  future  upgrading,  ^e  five  major  limitations  include: 
(1)  the  limited  number  of  training  devices  available,  (2)  the 
unavailability  of  certain  types  of  training  devices,  (3)  the  in¬ 
ability  to  impact  the  design  of  training  devices  because  of  the 
exceptionally  long  procurement  cycle,  (4)  the  use  of  less  than 
optimal  devices,  and  (5)  the  limited  range  of  media  available  for 
implementation  in  the  F-16  training  system.  ^ 

The  report  identifies  four  conditions  that  acted  to  pre¬ 
dispose  the  media  selection  process  to  the  constraints  identified 
above.  First,  the  F-16  syllabus  had  to  be  constructed  to  meet 
preexisting  aircraft  use  level  expectations  instead  of  allow¬ 
ing  instructional  considerations  to  drive  the  use  level.  Second, 
major  training  devices  were  designed  and  procured  before  specific 
training  uses  for  those  devices  were  generated.  Third,  no  in¬ 
structional  design  expertise  was  employed  during  the  initial 
stages  of  training  device  design  and  procurement.  Finally, 
adequate  funds  and  command  attention  were  not  directed  to  the 
procurement  of  major  F-16  training  devices. 

Based  upon  these  constraints  and  limitations,  the  report 
makes  six  specific  recommendations:  (1)  establish  and  execute  a 
plan  for  system  growth  that  will  eventually  minimize  these  con¬ 
straints,  (2)  design  simulated  practice  environments  an  scenarios 
that  take  fullest  advantage  of  the  capabilities  of  existing 
training  devices,  (3)  maintain  and  update  the  data-based  syllabus 
created  during  the  developnent  phase  and  utilize  feedback  from 
evaluations  to  adjust  the  syllabus,  (4)  examine  the  training 
functions  allocated  to  each  training  device  to  consider  possible 
reassignment  to  other  media,  (5)  employ  state-of-the-art  training 
techniques  to  increase  the  efficiency  of  training  device  utiliza¬ 
tion,  and  (6)  carefully  monitor  training  device  use  and  availa¬ 
bility  to  ensure  the  fullest  use  of  training  device  time. 
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TRAINING  MEDIA  CONSTRAINTS 
AND  LIMITATIONS 


1.0  PURPOSE  AND  ORGANIZATION 


The  purpose  of  this  report  is  to  identify  the  constraints 
that  affected  the  media  recommended  for  use  in  the  F-16  training 
system.  The  purposes  for  doing  this  are:  (1)  to  identify  the 
limitations  associated  with  the  media  selection  efforts  for  the 
present  F-16  training  system  so  that  future  system  decisions  can 
be  oriented  toward  their  removal,  and  (2)  to  identify  areas  of 
potential  growth  within  the  F-16  training  system  which  will 
become  feasible  as  existing  constraints  are  removed  by  time. 

Media  recommendations  for  the  F-16  training  system  are  made 
in  project  report  no.  31,  "F-16  Training  Media  Mix."  Other 
reports  discussing  the  media  selection  process  itself  include  no. 
30,  "F-16  Training  System  Media  Report,"  no.  32,  "F-16  Training 
Media  Support  Requirements,"  and  no.  33,  "F-16  Training  Media 
Constraints  and  Limitations." 

Media  selection  is  the  process  of  maximizing  the  benefits 
obtained  from  training  media  while  minimizing  the  cost.  More 
than  choosing  the  least  expensive  media,  media  selection  is  a 
process  of  balancing  capability  and  cost  along  with  several  other 
factors  to  obtain  the  best  media  mix.  The  full  range  of  factors 
considered  in  the  process  includes  such  things  as  cost,  lead  time 
for  procurement,  requirement  for  specialized  procurements,  main¬ 
tenance  requirements  in  terms  of  facilities  and  personnel,  opera¬ 
tional  expense,  difficulty  of  revision,  existence  of  required 
production  support  facilities,  and  several  other  factors. 

A  constraint  can  be  placed  upon  media  selection  because  of 
existing  conditions  in  any  of  these  areas.  Adequate  production 
facilities  may  not  exist,  costs  may  be  too  high  for  current  bud¬ 
gets,  or  lead  times  required  to  ready  media  for  use  may  be  too 
long  for  short-range  use.  Designs  are  often  created  by  instruc¬ 
tional  designers  which  require  sophisticated  media  to  execute 
sophisticated  training  routines,  but  in  some  cases  constraints  do 
not  allow  the  use  of  those  media  devices.  In  those  cases  the 
designer  acknowledges  the  real  world  by  adjusting  the  training 
plan  to  accommodate  the  less  than  optimal  training  environment. 

Constraints  affecting  media  selections  usually  change  with 
time.  New  sources  of  support  may  be  constructed  or  identified 
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through  reorganization,  operating  costs  may  be  reduced,  and  once 
expensive  systems  of  media  may  become  relatively  less  expensive 
as  other  training  resource  costs  rise.  As  this  happens,  the 
original  plans  of  the  designer  become  attainable,  and  the  system 
may  be  modified  to  operate  more  effectively. 

In  the  design  of  the  F-16  training  system  several  con¬ 
straints  were  operating  as  media  were  selected,  and  the  final 
media  selections  reflect  the  best  accommodation  to  those  existing 
conditions.  This  report  contains  recommendations  and  references 
to  plans  which  are  intended  to  diminish  the  effects  of  the  con¬ 
straints  experienced  in  the  F-16  media  selection  process.  Some 
of  the  constraints  acting  on  the  F-16  selections  are  removable, 
and  this  report  references  plans  which  have  been  made  for  the 
growth  of  the  system  as  they  are  taken  away.  Other  constraints 
which  affected  media  selections  were  the  result  of  institutional 
and  timing  factors  which  caused  decisions  to  be  made  which  will 
not  be  easily  reversed.  Recognition  of  those  factors  and  plans 
for  dealing  with  them  are  also  referenced  in  this  report. 

This  report  is  organized  to  present  the  limitations  of  the 
current  F-16  media  selections  and  the  factors  that  have  influ¬ 
enced  them.  Section  2.0  names  the  limitations.  Section  3.0 
describes  the  conditions  which  governed  the  media  selections  and 
gave  rise  to  the  limitations.  Section  4.0  describes  the  result 
and  recommends  approaches  for  dealing  with  these  constraints  in 
the  future. 

Discussions  in  this  report  which  refer  to  instructional 
media  should  be  understood  to  refer  to  both  academic  media 
presentation  devices  and  simulation  training  devices,  which  in 
many  cases  attempt  to  represent  physical  properties  of  the 
cockpit . 


2.0  F-16  MEDIA  CONSTRAINTS  AND  LIMITATIONS 


The  constraints  and  limitations  inherent  in  F-16  media 
selections  include  limitations  on  (1)  the  instructional  strate¬ 
gies  which  may  be  implemented  with  the  student  and  (2)  the  range 
of  instructional  and  simulation  experiences  which  may  be  provided 
under  the  present  media  selections.  These  constraints  arise  for 
several  reasons: 

1.  Number  of  devices:  Some  constraints  are  experienced  due 
to  the  number  of  training  devices  in  which  extensive 
training  experience  is  desired  but  not  possible.  Exam¬ 
ples  of  devices  which  are  constrained  in  this  way  are 
the  aircraft  and  the  Operational  Flight  Trainer  (OFT). 
Constraints  on  these  two  devices  serve  as  the  major 
limiting  factor  in  the  design  of  the  F-16  training 
system.  Not  only  are  media  decisions  affected,  but 
management,  scheduling,  and  syllabus  decisions  as  well. 
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2. 


Unavailable  devices:  Some  instructional  media  devices 
are  unavailable  for  use  in  F-16  training  program  due  to 
cost  and  time  factors.  Presentation  devices  automated 
by  computer,  such  as  computer-assisted  instruction,  are 
among  the  unavailable  media  recommended  for  F-16  train¬ 
ing.  A  second  class  of  devices  includes  the  part-task 
trainer.  This  is  a  computer-driven  simulation  of  the 
operation  of  certain  complex  physical  systems  of  the 
aircraft,  such  as  the  Stores  Management  Set  (SMS)  and 
the  Fire  Control/Navigation  panel  (FCNP) ,  not  only  as 
isolated  systems,  but  in  interaction  as  well. 

3.  Early  procurement  of  devices:  Recommendations  for  the 
modification  and  use  of  some  of  the  F-16  training 
devices  have  been  made  in  the  course  of  the  F-16  train¬ 
ing  development  project.  For  example,  development 
report  no.  22,  "Recommendations  for  F-16  Operational 
Flight  Trainer  (OFT)  Design  Improvements,"  recommends  14 
separate  modifications  to  the  OFT  that  would  make  it 
more  effective  instruc tionally .  At  present  there  are  no 
definite  plans  to  implement  these  recommendations. 

4.  Less  than  optimal  devices:  In  some  cases  instructional 
devices  procured  for  use  in  F-16  training  were  not 
reviewed  by  instructional  design  personnel  for  instruc¬ 
tional  utility.  The  concept  behind  some  of  these 
devices  was  sound,  but  the  design  lacked  some  features 
which  would  make  the  device  more  capable  instruction- 
ally.  For  example,  both  the  Cockpit  procedures  Trainer 
(CPT)  and  Egress  proceduress  Trainer  (EPT)  were  designed 
prior  to  review  by  the  instructional  design  team. 

5.  Limited  media:  Since  computer-assisted  instruction 
(CAI)  and  part-task  trainers  were  unavailable  due  to  ,, 
cost  and  time  factors,  a  practical  rather  than  ideal 
approach  to  media  selection  was  used.  This  approach 
emphasized  mixing  those  media  that  were  available  into 
an  acceptable  instruction  system.  The  unavailability  of 
media  has  probably  caused  some  compromise  in  the  quality 
of  instruction,  but  the  resulting  system  is  well  within 
reasonable  tolerances  in  terms  of  instructional  adequa¬ 
cy.  It  is  hoped,  however,  that  future  considerations 
will  include  the  possibility  of  using  CAI. 

As  a  result  of  the  factors  listed  above,  the  media  selection 
for  the  F-16  training  system  did  not  include  the  full  range  of 
available  media.  From  a  theoretical  perspective,  this  selection 
is  less  than  ideal,  but  the  realities  of  real  world  constraints 
often  dictate  instructional  development  outcomes.  This  should 
not  stop  future  efforts  aimed  at  improving  the  system. 


3.0  CONDITIONS  AND  PRACTICES  ACTING  AS  PRECURSORS  TO  MEDIA 
CONSTRAINTS 


This  section  identifies  the  main  factors  operating  in  the 
determination  of  F-16  media  constraints.  The  following  four 
problems  impact  F-16  media  capabilities: 

1.  Necessity  of  constructing  F-16  syllabi  to  meet  preexist¬ 
ing  aircraft  use  level  expectations:  It  is  a  common  but 
less  than  optimal  situation  in  aircrew  training  that  the 
major  syllabus  parameters  are  set  far  in  advance  of 
determining  what  must  be  trained.  Instructional  devel¬ 
opers  for  aircrew  training  are  consistently  asked  to 
build  the  "optimal"  syllabus  but  to  do  it  within  "no 
more  than  x  number"  of  rides.  This  is  a  reversal  of  the 
logical  order  of  development.  To  determine  how  much 
flight  time  will  be  used  in  training  it  is  more  logical 
to  first  determine  what  will  be  instructed  and  how. 
Nonetheless,  it  has  become  standard  practice  to  state 
the  amount  of  flight  time  available  and  build  the 
syllabus  around  that  value. 

2.  Ordering  of  training  devices  without  consideration  of 

specific  training  requirements  or  concomitant  volume  of 
use:  The  long  lead  time  required  for  the  procurement  of 

training  devices  often  requires  them  to  be  ordered  way 
in  advance  of  the  determination  of  exact  training 
requirements.  This  again  is  a  reversal  of  the  logical 
order  of  development  processes  and  works  against  the 
design  of  training  devices  matched  to  training  needs. 
Most  important  in  this  respect,  it  forces  the  procure¬ 
ment  of  training  devices  based  on  subjective  judgements 
and  traditional  device  procurement  patterns. 

3.  Omission  of  instructional  design  expertise  from  initial 

stages  of  training  system  and  training  device  design  and 
procurement:  The  problems  described  in  the  previous 

paragraph  related  to  the  early  ordering  of  training 
devices  are  compounded  by  the  normal  practice  of  not 
involving  instructional  designers  in  the  training  device 
design  and  procurement  process.  On  the  F-16  project, 
more  than  on  most  projects  for  aircrew  training  develop¬ 
ment,  the  instructional  designer  did  play  an  active  part 
in  this  process,  but  only  in  the  very  terminal  stages  of 
device  design,  and  only  for  one  device,  the  OFT.  Even 
at  early  stages  of  weapon  system  development,  there  is 
enough  data  available  concerning  weapon  characteristics 
and  missions  to  make  some  generalizations  about  the 
training  burden,  and  device  designs  which  normally  begin 
to  evolve  at  this  stage  could  profit  from  what  techni¬ 
ques  an  instructional  designer  could  bring  to  bear  on 
the  device  design  problem.  Techniques  for  achieving 
this  and  specific  timelines  and  design  decisions  are 
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described  in  "Fleet  Aviation  Instructional  Systems 
Development  Model:  For  Emerging  Weapon  Systems,"  a 
report  prepared  for  the  US  Navy  Training  Equipment 
Center  under  contract  no.  N61339-76-C-0062. 

4.  Allocation  of  funds  and  attention  to  development  of  the 
training  program  and  its  training  devices:  For  the  F-16 
program  more  than  most  other  programs  an  attempt  has 
been  made  to  allocate  adequate  funds  and  command  atten¬ 
tion  to  the  development  of  a  training  system.  Though 
there  have  been  many  lessons  learned  in  this  attempt  to 
interface  contractor  and  military  personnel,  there  has 
been  much  success  attributable  to  the  proper  funding  and 
proper  management  of  many  areas  of  the  project.  Even 
with  these  successes,  there  have  been  some  areas  in 
which  better  funding  or  better  management  of  the  devel¬ 
opment  processes  would  have  netted  additional  benefits 
and  efficiencies  for  the  training  system.  Proper  design 
of  an  entire  range  of  training  devices  needed,  for 
instance,  is  an  area  in  which  the  F-16  project  could 
have  improved.  This  range  of  training  devices,  balanced 
to  obtain  maximum  effective  use  of  all  devices  at  mini¬ 
mum  overall  cost,  would  include  not  only  simulation 
media,  but  academic  instructional  media  as  well. 


4.0  RECOMMENDATIONS  FOR  MINIIIIZING  CONSTRAINT  EFFECTS 


This  section  makes  recommendations  for  minimizing  the 
effects  of  constraints  upon  the  F-16  system.  The  main 
recommendation  centers  on  removing  the  constraints  through 
appropriate  levels  of  funding  for  system  improvements  over  a 
period  of  time.  Other  recommendations  deal  with  policies  which 
may  be  implemented  independent  of  the  decision  to  improve  the 
system  through  growth.  The  recommendations  follow  below. 

Recommendation  1:  Follow  a  plan  of  system  growth  which 
would  eventually  reduce  constraints.  A  plan  for  system  growth  in 
both  instructional  and  management  capabilities  has  been  issued  in 
project  report  no.  27,  "F-16  Instructional  System  Design  Alterna¬ 
tives."  That  report  recommends  a  series  of  five  major  steps  of 
system  growth  which  would  remove  the  majority  of  the  constraints 
identified  in  this  report,  including  constraints  on  the  range  of 
training  devices  available  for  use  and  limits  on  the  amount  of 
time  available  for  use  on  more  expensive  devices.  Easing  the 
limitations  of  time  available  on  expensive  devices,  of  course, 
would  be  in  some  ways  analogous  to  obtaining  additional  copies  of 
training  devices.  Instructional  strategy  constraints  would  be 
removed  for  both  academic  media  and  for  the  use  of  training 
devices  in  simulations  so  far  as  training  device  capabilites 
would  allow  it.  Constraints  remaining  would  be  those  relative  to 
the  original  design  of  the  training  devices  for  instructional 
functions.  Shortcomings  in  training  devices  already  procured 
would  not  be  removed. 
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Recommendation  2;  Pay  additional  attention  to  the  design  of 
simulated  practice  environments  and  scenarios  that  use  to  the 
fullest  extent  the  capabilities  of  existing  training  devices.  If 
new  devices  with  better  instructional  capabilities  can  not  be 
procured  for  F-16  use,  it  may  be  possible  to  derive  additional 
instructional  benefit  from  utilization  of  existing  devices  in 
novel  ways.  in  most  cases  this  would  involve  the  creation  of 
simulations  which  were  less  and  less  equipment-centered  and  more 
and  more  student  and  mental-rehearsal  centered.  Research  has 
demonstrated  the  effectiveness  of  mental  simulation  carried  out 
through  mental  rehearsal  techniques.  Even  simulations  which  have 
equipment  related  to  them  can  involve  an  element  of  mental 
rehearsal.  It  is  not  unreasonable  to  look  at  the  amount  of 
environment  supplied  mentally  by  the  student  during  an  exercise 
as  a  major  dimension  of  the  idea  of  a  simulation. 

Recommendation  3:  Carefully  maintain  the  data-based 
llabus  created  during  development  and  utilize  feedback  from 
evaluations  to  adjust  the  syllabus  for  efficiency.  The  syllabus 
constructed  for  F-16  B/C  course  training  was  built  according  to 
principles  that  insured  (1)  that  every  event  included  in  every 
sortie  was  justified  as  a  required  training  event,  (2)  that 
sorties  did  not  include  more  than  what  could  be  accomplished 
during  the  time  slotted,  and  (3)  that  adequate  time  was  reserved 
for  each  event.  Underlying  the  syllabus  is  a  data  bank  which 
specifies  for  each  task  to  be  learned  the  best  estimate  which 
could  be  made  of  the  number  of  demonstrations,  practices,  and 
evaluations  required  by  the  typical  student  to  learn  to  perform 
the  task.  The  syllabus  is  a  reflection  of  this  data.  As  exper¬ 
ience  in  teaching  students  accumulates,  the  original  estimates 
used  to  build  the  syllabus  will  be  either  verified  or  disputed. 
The  data  to  do  this  will  come  through  the  gradeslip,  since 
entries  are  to  be  made  on  F-16  gradeslips  of  actual  numbers  of 
practices  of  each  event.  When  it  is  found  that  the  practice 
jmount  for  a  particular  task  was  overestimated,  the  appropriate 
/eduction  can  take  place  in  the  amount  of  syllabus  time  allotted 
to  the  task.  An  effort  will  be  made  to  "push  down"  training 
:asks  to  less  and  less  expensive  media  without  compromising 
instructional  effectiveness.  Underestimates,  of  course,  must  be 
handled  by  adding  the  appropriate  amount  of  practice.  If  the 
syllabus  is  properly  maintained,  there  is  a  high  probability  that 
opportunities  for  economy  will  be  found  in  the  overestimations 
which  exist  within  the  syllabus. 

At  some  future  time,  as  the  F-16  management  system  is  com¬ 
puterized,  the  syllabus  process  also  could  be  computerized  to 
make  the  adjustment  of  the  syllabus  automatic,  based  on  current 
student  performances.  The  logical  extension  of  this  idea,  of 
course,  is  that  the  F-16  training  system  would  be  able  to  produce 
for  each  student  a  unique  syllabus,  including  media  selection, 
generated  by  computer  and  tailored  to  the  needs  and  learning  rate 
of  each  student.  The  exact  needs  of  each  student  could  be 
generated  through  manipulation  of  the  student's  own  performance 
data  and  the  syllabus  data  base.  Gradeslips  under  this  mode  of 
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operation  would  be  generated  prior  to  each  flight  for  the 
particular  student.  Since  this  method  of  syllabus  generation 
would  move  many  students  through  training  ahead  of  the  average 
pace,  it  can  be  anticipated  that  it  would  affect  important  and 
large  economies  in  training  costs  without  incurring  many  of  the 
negative  motivational  side-effects  of  other  plans  for  moving  fast 
students  through  training  more  rapidly.  Under  this  mode  of 
operation,  however,  it  would  be  important  to  see  to  it  that 
students  who  moved  through  training  rapidly  perceived  it  as 
beneficial  to  themselves  to  do  so.  Antimotivating  pressures 
would  otherwise  reduce  the  effectiveness  of  the  system. 

Recommendation  4;  Study  the  training  functions  allocated  to 
each  training  device  to  determine  possible  reassignment  of  train¬ 
ing  to  devices.  Because  the  F-16  syllabus  and  media  selections 
were  carried  out  by  experts  with  opinions,  it  is  possible  that  in 
some  cases  training  of  some  tasks  was  assigned  on  the  basis  of 
tradition  rather  than  on  the  basis  of  a  well-thoi^ht  instruc¬ 
tional  plan.  It  is  possible  that  some  functions  designated  for 
simulator  training  are  more  appropriate  only  for  the  aircraft,  or 
vice  versa.  Past  decisions  will  become  subject  to  review,  espe¬ 
cially  upon  delivery  of  the  OFT  for  training  and  as  instructors 
learn  what  can  and  cannot  be  adequately  practiced  in  it. 

In  a  reevaluation  to  be  completed  later  in  the  project, 
contractor  and  military  personnel  will  create  a  listing  of  the 
tasks  assigned  to  each  medium  for  instruction.  These  lists  will 
be  scanned  to  determine  possible  better  assignments  or  misassign- 
ments  of  tasks  to  training  media.  This  work  will  occur  under  the 
requirements  of  CDRL  B047,  "Ground  Based  Training  Limitations." 

In  addition,  evaluation  data  will  help  identify  those  areas  that 
need  better  assignments. 

Recommendation  5:  Employ  up-to-date  training  methods  where 
possible  to  increase  the  efficiency  of  training  time.  Several 
methods  which  are  not  currently  standard  practice  in  aircrew 
training  have  been  demonstrated  through  research  and  development 
efforts  to  be  effective  in  producing  skilled  performers  with  less 
training  resource  requirement  on  some  tasks.  Techniques 
researched  by  AF  Human  Resources  Laboratory,  Williams  AFB, 

Arizona  using  backward  chaining  in  simulator  instruction  have 
demonstrated  high  efficiency  rates.  The  employment  of  these 
techniques  first  on  a  test  basis  and  then  if  feasible  on  a  full- 
scale  basis  has  been  recommended  in  other  project  docunentation . 
It  is  expected  that  savings  of  training  device  time  could  be 
effected  by  doing  so. 

Recommendation  6;  Carefully  monitor  training  device  use  and 
availability  data  to  insure  the  fullest  use  of  training  device 
time.  The  formative  evaluation  subsystem  designed  as  part  of  the 
F-16  training  system  will  be  designed  to  collect  usage  data  on 
all  training  devices.  Training  devices  which  are  counted  in  the 
syllabus  but  not  available  because  of  inefficient  use  or  break¬ 
downs  are  costly  to  system  operation.  A  future  project  task  will 
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investigate  training  device  availability  (CDRL  B046>  "Training 
Device  Availability")  and  assess  its  impact  on  the  flow  of 
students  through  the  syllabus.  It  is  important  that  the  role  of 
the  evaluation  system  in  collecting  and  reporting  data  on  device 
usage  be  implemented  and  used  to  adjust  system  policy  and 
practices . 
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